Driving assistance device, driving assistance method, and storage medium

ABSTRACT

According to an embodiment, a driving assistance device includes a recognizer configured to recognize a surrounding situation of a vehicle and a steering controller configured to control at least steering of the vehicle so that the vehicle travels in a travel lane when the vehicle tends to deviate from the travel lane on the basis of a recognition result of the recognizer. The steering controller controls the steering of the vehicle by selecting any steering characteristic in accordance with information obtained from an occupant of the vehicle among a plurality of steering characteristics.

CROSS-REFERENCE TO RELATED APPLICATION

Priority is claimed on Japanese Patent Application No. 2022-100931,filed Jun. 23, 2022, the content of which is incorporated herein byreference.

BACKGROUND Field of the Invention

The present invention relates to a driving assistance device, a drivingassistance method, and a storage medium.

Description of Related Art

In recent years, efforts to provide access to sustainable transportsystems that take into account vulnerable groups among trafficparticipants have become active. To implement these, research anddevelopment for further improving traffic safety and convenience throughresearch and development related to driving assistance technology arebeing focused on. In relation to this, in the related art, technologyfor applying reaction force torque to a steering mechanism so that avehicle does not deviate from a target travel line in a lane keepingassistance system (LKAS) is disclosed (for example, Japanese UnexaminedPatent Application, First Publication No. 2012-232704).

SUMMARY

Meanwhile, a process of controlling reaction force torque in accordancewith an occupant's preference has not been taken into account in theconventional driving assistance technology. Thus, there is a problemthat it may be difficult to provide appropriate assistance to anoccupant during the execution of lane keeping assistance.

To solve the above-described problem, an objective of this applicationis to provide a driving assistance device, a driving assistance method,and a storage medium capable of providing appropriate lane keepingassistance for an occupant. Also, it is possible to contribute to thedevelopment of a sustainable transportation system.

A driving assistance device, a driving assistance method, and a storagemedium according to the present invention adopt the followingconfigurations.

-   -   (1): According to an aspect of the present invention, there is        provided a driving assistance device including: a recognizer        configured to recognize a surrounding situation of a vehicle;        and a steering controller configured to control at least        steering of the vehicle so that the vehicle travels in a travel        lane when the vehicle tends to deviate from the travel lane on        the basis of a recognition result of the recognizer, wherein the        steering controller controls the steering of the vehicle by        selecting any steering characteristic in accordance with        information obtained from an occupant of the vehicle among a        plurality of steering characteristics.    -   (2): In the above-described aspect (1), the plurality of        steering characteristics include at least a first steering        characteristic and a second steering characteristic having a        larger steering reaction force of the vehicle than the first        steering characteristic.    -   (3): In the above-described aspect (2), the second steering        characteristic has a larger steering reaction force        corresponding to a distance from a center of the travel lane of        the vehicle than the first steering characteristic.    -   (4): In the above-described aspect (1), the steering controller        controls the steering of the vehicle by switching the steering        characteristic to the steering characteristic selected by the        occupant among the plurality of steering characteristics.    -   (5): In the above-described aspect (1), the driving assistance        device further includes a driving characteristic acquirer        configured to acquire driving characteristics of the vehicle        from the occupant of the vehicle, wherein the steering        controller controls the steering of the vehicle by selecting any        steering characteristic among the plurality of steering        characteristics on the basis of the driving characteristics        acquired by the driving characteristic acquirer.    -   (6): In the above-described aspect (2), the plurality of        steering characteristics further include a third steering        characteristic for controlling the steering of the vehicle        according to a steering reaction force between the steering        reaction force based on the first steering characteristic and        the steering reaction force based on the second steering        characteristics, and the steering controller temporarily        controls the steering of the vehicle according to the third        steering characteristic when the steering characteristic is        switched between the first steering characteristic and the        second steering characteristic.    -   (7): In the above-described aspect (1), the driving assistance        device further includes an occupant state acquirer configured to        acquire a state of the occupant of the vehicle, wherein the        steering controller controls the steering of the vehicle by        selecting any steering characteristic among the plurality of        steering characteristics on the basis of the occupant state        acquired by the occupant state acquirer.    -   (8): In the above-described aspect (1), the steering controller        controls the steering of the vehicle by selecting any steering        characteristic among the plurality of steering characteristics        on the basis of information obtained from the occupant of the        vehicle and a road situation near the vehicle recognized by the        recognizer.    -   (9): According to another aspect of the present invention, there        is provided a driving assistance method including: recognizing,        by a computer, a surrounding situation of a vehicle;        controlling, by the computer, at least steering of the vehicle        so that the vehicle travels in a travel lane when the vehicle        tends to deviate from the travel lane on the basis of a        recognition result; and controlling, by the computer, the        steering of the vehicle by selecting any steering characteristic        in accordance with information obtained from an occupant of the        vehicle among a plurality of steering characteristics.    -   (10): According to yet another aspect of the present invention,        there is provided a computer-readable non-transitory storage        medium storing a program for causing a computer to: recognize a        surrounding situation of a vehicle; control at least steering of        the vehicle so that the vehicle travels in a travel lane when        the vehicle tends to deviate from the travel lane on the basis        of a recognition result; and control the steering of the vehicle        by selecting any steering characteristic in accordance with        information obtained from an occupant of the vehicle among a        plurality of steering characteristics.

According to the above-described aspects (1) to (10), it is possible toprovide appropriate lane keeping assistance for an occupant.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a configuration diagram of a vehicle system to which a drivingassistance device according to an embodiment is applied.

FIG. 2 is a diagram for describing a function of a steering controller.

FIG. 3 is a diagram for describing a plurality of steeringcharacteristics according to the embodiment.

FIG. 4 is a diagram for describing the positioning of the reaction forcein each of a plurality of steering characteristics.

FIG. 5 is a flowchart showing an example of a flow of a process executedby the driving assistance device of the embodiment.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of a driving assistance device, a drivingassistance method, and a storage medium of the present invention will bedescribed with reference to the drawings.

[Overall Configuration]

FIG. 1 is a configuration diagram of a vehicle system 1 to which adriving assistance device according to the present embodiment isapplied. A vehicle (hereinafter referred to as a vehicle M) in which thevehicle system 1 is mounted is, for example, a vehicle such as atwo-wheeled vehicle, a three-wheeled vehicle, or a four-wheeled vehicle,and a drive source thereof is an internal combustion engine such as adiesel engine or a gasoline engine, an electric motor, or a combinationthereof. The electric motor operates using electric power generated by apower generator connected to the internal combustion engine or electricpower that is supplied when a secondary battery or a fuel cell isdischarged. In the following description, it is assumed that the vehicleM is a four-wheeled vehicle. Hereinafter, as an example, the drivingassistance device may be applied to an automated driving vehicle. Inautomated driving, for example, driving control is executed byautomatically controlling one or both of steering andacceleration/deceleration of the vehicle M. The driving control of thevehicle M may include, for example, various types of driving assistancesuch as adaptive cruise control (ACC), an LKAS, lane change assist(LCA), a forward collision warning (FCW), and a collision mitigationbraking system (CMBS). In the automated driving vehicle, partial ortotal driving may be controlled according to manual driving of anoccupant (a driver).

For example, the vehicle system 1 includes a camera (an example of animager) 10, a radar device 12, a light detection and ranging (LIDAR)sensor 14, a physical object recognition device 16, a communicationdevice 20, a human machine interface (HMI) 30, a vehicle sensor 40, anavigation device 50, a driver monitor camera (an example of a cabinimager) 70, driving operation elements 80, an driving assistance device100, a travel driving force output device 200, a brake device 210, and asteering device 220. Such devices and equipment are connected to eachother by a multiplex communication line such as a controller areanetwork (CAN) communication line, a serial communication line, or awireless communication network. The configuration shown in FIG. 1 ismerely an example and some of the components may be omitted or othercomponents may be further added. A combination of the camera 10, theradar device 12, and the LIDAR sensor 14 is an example of an “externalsensor ES.” The external sensor ES may include the physical objectrecognition device 16 or may include other detectors (for example, soundnavigation and ranging (SONAR)) that recognize the surrounding situationof the vehicle M. The external sensor ES may have a configuration ofonly the camera 10 or a simple configuration of only the camera 10 andthe radar device 12. The HMI 30 is an example of a “receiver” and an“output.”

For example, the camera 10 is a digital camera using a solid-stateimaging element such as a charge-coupled device (CCD) or a complementarymetal-oxide semiconductor (CMOS). The camera 10 is attached to anylocation on the vehicle M. For example, when the view in front of thevehicle M is imaged, the camera 10 is attached to an upper part of afront windshield, a rear surface of a rearview mirror, or the like. Whenthe view to the rear of the vehicle M is imaged, the camera 10 isattached to an upper part of a rear windshield, a back door, or thelike. When the views to the side and the rear side of the vehicle M areimaged, the camera 10 is attached to a door mirror or the like. Forexample, the camera 10 periodically and iteratively images thesurroundings of the vehicle M. The camera 10 may be a stereo camera.

The radar device 12 radiates radio waves such as millimeter waves aroundthe vehicle M and detects at least a position (a distance to and adirection) of a physical object by detecting radio waves (reflectedwaves) reflected by the physical object. The radar device 12 is attachedto any location on the vehicle M. The radar device 12 may detect aposition and a speed of the physical object in a frequency modulatedcontinuous wave (FM-CW) scheme.

The LIDAR sensor 14 radiates light (or electromagnetic waves having awavelength close to light) to the vicinity of the vehicle M and measuresscattered light. The LIDAR sensor 14 detects a distance from an objecton the basis of time from light emission to light reception. Theradiated light is, for example, pulsed laser light. The LIDAR sensor 14is attached to any location on the vehicle M.

The physical object recognition device 16 performs a sensor fusionprocess on detection results from some or all of the components includedin the external sensor ES to recognize a position, a type, a speed, andthe like of a physical object near the vehicle M. The physical objectrecognition device 16 outputs recognition results to the drivingassistance device 100. The physical object recognition device 16 mayoutput detection results of the external sensor ES to the drivingassistance device 100 as they are. The physical object recognitiondevice 16 may be omitted from the vehicle system 1.

The communication device 20 communicates with another vehicle located inthe vicinity of the vehicle M using, for example, a cellular network ora Wi-Fi network, Bluetooth (registered trademark), dedicated short rangecommunication (DSRC), or the like, or communicates with various types ofserver devices via a radio base station.

The HMI 30 presents various types of information to an occupant of thevehicle M and receives an input operation from the occupant undercontrol of the HIM controller 150. The HMI 30 includes, for example, adisplay device, a speaker, a microphone, a buzzer, a touch panel, keys,and the like. The HMI 30 includes, for example, a switch (a drivingassistance changeover switch) for accepting whether or not to executevarious types of driving assistance such as an LKAS and ACC and a switchfor switching and adjusting a steering characteristic during theexecution of the LKAS. The steering characteristics include, forexample, a characteristic related to a weight of the steering wheeloperated by the occupant or a characteristic related to a magnitude of areaction force of the steering wheel generated to return into the travellane when the vehicle M tends to deviate from the travel lane in controlbased on the LKAS (hereinafter referred to as “LKAS control”). Detailsof the steering characteristics in the embodiment will be describedbelow.

The vehicle sensor 40 includes a vehicle speed sensor configured todetect the speed of the vehicle M, an acceleration sensor configured todetect acceleration, a yaw rate sensor configured to detect an angularspeed around a vertical axis, a direction sensor configured to detect adirection of the vehicle M, and the like. The vehicle sensor 40 mayinclude a steering angle sensor configured to detect a steering angle ofthe vehicle M (which may be an angle of the steering wheel or anoperating angle of the steering wheel). The vehicle sensor 40 mayinclude a position sensor configured to acquire the position of thevehicle M. The position sensor is, for example, a sensor configured toacquire position information (longitude/latitude information) from aGlobal Positioning System (GPS) device. The position sensor may be asensor configured to acquire position information using a globalnavigation satellite system (GNSS) receiver 51 of the navigation device50.

For example, the navigation device 50 includes a global navigationsatellite system (GNSS) receiver 51, a navigation HMI 52, and a routedecider 53. The navigation device 50 holds map information 54 in astorage device such as a hard disk drive (HDD) or a flash memory. TheGNSS receiver 51 identifies a position of the vehicle M on the basis ofa signal received from a GNSS satellite. The position of the vehicle Mmay be identified or corrected by an inertial navigation system (INS)using an output of the vehicle sensor 40. The navigation HMI 52 includesa display device, a speaker, a touch panel, keys, and the like. Thenavigation HMI 52 may be partly or wholly shared with theabove-described HMI 30. For example, the route decider 53 decides on aroute (hereinafter referred to as a route on a map) from the position ofthe vehicle M identified by the GNSS receiver 51 (or any input position)to a destination input by the occupant using the navigation HMI 52 withreference to the map information 54. The map information 54 is, forexample, information in which a road shape is expressed by a linkindicating a road and nodes connected by the link. The map information54 may include the curvature of a road, point of interest (POI)information, and the like. The map information 54 may include, forexample, lane center information, lane boundary information (roadmarkings), a lane width, and the like or may include road information(for example, expressways and general roads), traffic regulationinformation (for example, a speed limit), address information (anaddress/postal code), facility information, telephone numberinformation, and the like. The map information 54 may be updated at anytime by the communication device 20 communicating with other devices.The map information 54 may be stored in the storage of the drivingassistance device 100 to be described below.

The navigation device 50 may provide route guidance using the navigationHMI 52 on the basis of the route on the map. The navigation device 50may be implemented, for example, according to a function of a terminaldevice such as a smartphone or a tablet terminal possessed by theoccupant. The navigation device 50 may transmit a current position and adestination to a navigation server via the communication device 20 andacquire a route equivalent to the route on the map from the navigationserver.

The driver monitor camera 70 is, for example, a digital camera that usesa solid-state image sensor such as a CCD or a CMOS. The driver monitorcamera 70 is attached to any location on the vehicle M with respect to aposition and a direction where the head of the occupant (hereinafter,the driver) sitting in the driver's seat of the vehicle M can be imagedfrom the front (in a direction in which his/her face is imaged). Forexample, the driver monitor camera 70 is attached to an upper part of adisplay device provided on the central portion of the instrument panelof the vehicle M. The driver monitor camera 70 acquires images obtainedby imaging the cabin including the occupant of the vehicle M from anarrangement position thereof at prescribed intervals. The driver monitorcamera 70 may acquire images including a fellow passenger at prescribedintervals. The driver monitor camera 70 may acquire a sound within thecabin (for example, a conversation of an occupant) or the like inaddition to the images. Information acquired by the driver monitorcamera 70 is output to the driving assistance device 100.

For example, the driving operation elements 80 include an acceleratorpedal, a brake pedal, a shift lever, and other operation elements inaddition to a steering wheel. A sensor configured to detect an amount ofoperation or the presence or absence of an operation is attached to thedriving operation element 80 and a detection result of the sensor isoutput to the driving assistance device 100 or some or all of the traveldriving force output device 200, the brake device 210, and the steeringdevice 220. The steering wheel is an example of an “operation elementthat receives a steering operation by the driver.” The operation elementdoes not necessarily have to be annular, and may be in the form of avariant steering, a joystick, a button, or the like. The drivingoperation element 80 outputs a steering angle and an amount of steeringtorque when the occupant (driver) of the vehicle M steers the steeringwheel in a prescribed direction to the driving assistance device 100.

The driving assistance device 100 includes, for example, a recognizer110, a determiner 120, an acquirer 130, a driving controller 140, an HMIcontroller 150, and a storage 160. The recognizer 110, the determiner120, the acquirer 130, the driving controller 140, and the HMIcontroller 150 are implemented, for example, by a hardware processorsuch as a CPU executing a program (software). Some or all of thesecomponents may be implemented by hardware (including a circuit;circuitry) such as a large-scale integration (LSI) circuit, anapplication-specific integrated circuit (ASIC), a field-programmablegate array (FPGA), or a graphics processing unit (GPU) or may beimplemented by software and hardware in cooperation. The program may beprestored in a storage device (a storage device including anon-transitory storage medium) such as a hard disk drive (HDD) or aflash memory of the driving assistance device 100 or may be stored in aremovable storage medium such as a digital video disc (DVD) or a compactdisc-read-only memory (CD-ROM) and installed in the HDD or the flashmemory of the driving assistance device 100 when the storage medium (anon-transitory storage medium) is mounted in a drive device. The HMIcontroller 150 is an example of an “output controller.”

The storage 160 may be implemented by the above-described variousstorage devices, a solid-state drive (SSD), an electrically erasableprogrammable read only memory (EEPROM), a read-only memory (ROM), arandom-access memory (RAM), or the like. The storage 160 stores, forexample, steering characteristic information 162, programs, variousother types of information, and the like. The steering characteristicinformation 162 is, for example, characteristic information about areaction force generated in an operation direction of the steering wheelof the occupant when the vehicle M tends to deviate from the travel laneat the time of execution of LKAS control. The characteristic informationincludes information about a magnitude of the reaction force. Thesteering characteristic information 162 stores a plurality of presetsteering characteristics. Each of the plurality of steeringcharacteristics may be appropriately adjusted according to the settingof the occupant. The storage 160 may store, for example, drivingcharacteristic information of the occupant, occupant state information,and map information 54 to be described below.

The recognizer 110 recognizes a surrounding situation of the vehicle Mon the basis of information input from the external sensor ES. Forexample, the recognizer 110 recognizes states such as a position,velocity, and acceleration of a physical object located within aprescribed distance from the vehicle M on the basis of the informationinput from the external sensor ES. The physical object is, for example,a traffic participator such as another vehicle, a bicycle, or apedestrian. For example, the position of the physical object isrecognized as a position on absolute coordinates with a representativepoint (a center of gravity, a driving shaft center, or the like) of thevehicle M as the origin and is used for control. The position of thephysical object may be represented by a representative point such as acenter of gravity or a corner of the physical object or may berepresented by an area. The “state” of a physical object may includeacceleration or jerk of the physical object or an “action state” (forexample, whether or not a lane change is being made or intended). Therecognizer 110 may recognize types of physical objects (other vehicles,bicycles, and pedestrians), and the like on the basis of characteristicinformation such as sizes, shapes, and colors of the physical objects.

The recognizer 110 recognizes, for example, a lane in which the vehicleM is traveling (a travel lane). For example, the recognizer 110recognizes road markings (hereinafter referred to as “markings”) on theleft and right of the vehicle M from a camera image captured by thecamera 10 and recognizes the travel lane on the basis of positions ofthe recognized markings. The recognizer 110 may recognize the travellane by recognizing targets (a runway boundary and a road boundary) foridentifying lane positions including a shoulder, a curb, a median, aguardrail, a fence, a wall, and the like as well as the markings. Inthis recognition, the position of the vehicle M acquired from thenavigation device 50 and the processing result of the INS may be takeninto account. The recognizer 110 may recognize the travel lane withreference to the map information 54 on the basis of the positioninformation of the vehicle M obtained from the vehicle sensor 40 or mayrecognize the travel lane by comparing a pattern of the road markingobtained from the map information 54 (for example, an arrangement ofsolid lines and dashed lines) with a pattern of the road marking nearthe host vehicle M recognized from the image captured by the camera 10.The recognizer 110 may recognize adjacent lanes adjacent to the travellane of the vehicle M, temporary stop lines, obstacles, red lights, tollbooths, and other road events.

When the travel lane is recognized, the recognizer 110 recognizes aposition and orientation of the vehicle M for the travel lane. Forexample, the recognizer 110 may recognize the deviation of the referencepoint (a center or a center of gravity) of the vehicle M (the distancein a road width direction) from the center of the lane and the angleformed between the traveling direction of the vehicle M and the lineconnected to the lane center as the relative position and orientation ofthe vehicle M for the travel lane. Instead of this, the recognizer 110may recognize the position of the reference point of the vehicle M forany side end of the travel lane (the marking or the road boundary) orthe like as a position of the vehicle M relative to the travel lane.

For example, the recognizer 110 implements a function based onartificial intelligence (AI) and a function based on a previously givenmodel in parallel. For example, an “intersection recognition” functionmay be implemented by executing intersection recognition based on deeplearning or the like and recognition based on previously givenconditions (signals, road signs, or the like with which pattern matchingis possible) in parallel and performing comprehensive evaluation byassigning scores to both recognition processes.

The determiner 120 determines whether or not the vehicle M tends todeviate from the travel lane on the basis of the recognition result ofthe recognizer 110. For example, the determiner 120 acquires a distancebetween the center of the vehicle M (which may be the center of gravity)and the center of the travel lane of the vehicle M in a state in whichthe vehicle M is executing LKAS control and determines that the vehicleM tends to deviate from the travel lane when the acquired distance isgreater than or equal to a processing distance. The determiner 120 maydetermine that the vehicle M tends to deviate from the travel lane whenan amount of change (an increased amount) in a distance for a prescribedperiod of time is greater than or equal to a prescribed amount.

The determiner 120 may determine that the vehicle M tends to deviatefrom the travel lane when a distance from the marking dividing thetravel lane to the center of the vehicle M instead of the distance fromthe center of the travel lane is less than a prescribed distance or whena decrease amount of the distance for the prescribed period of time isgreater than or equal to a prescribed amount. The prescribed distancedescribed above may be a fixed distance or may be set to be variableaccording to a width of the travel lane, a width of the vehicle, or aroad type (for example, an expressway, a general road, or the like).

The acquirer 130 includes, for example, a driving characteristicacquirer 132 and an occupant state acquirer 134. The drivingcharacteristic acquirer 132 acquires the driving characteristics of thedriver on the basis of information detected by the vehicle sensor 40 andthe driving operation element 80. Driving characteristics include, forexample, a driving tendency indicating how the occupant drives thevehicle M. The driving tendency includes, for example, a tendency for aratio or a frequency at which the center position of the vehicle Mdeviates from the center of the lane to be high, a tendency that atiming of a steering operation is late (or early), a tendency for asteering amount to be small (or large), and the like when the drivingduration is greater than or equal to a prescribed period of time. Thedriving tendency may include a tendency for the brake timing to be late,a tendency to start suddenly when angry, a tendency to neglect checkingthe side mirror when talking to a fellow passenger, and the like. Thedriving characteristics may include, for example, information such asthe number of lane deviations and a degree of deviation when LKAScontrol is not performed and at the time of driving without right turns,left turns, and lane changes.

The driving characteristic acquirer 132 may designate informationacquired by the vehicle sensor 40 and the driving operation element 80as an input during the manual driving of the occupant and acquiredriving characteristics of the occupant (driver) using a trained modelfor outputting the driving characteristics. The trained model may beupdated using, for example, a function based on AI such as machinelearning (neural network) or deep learning using training (correct) dataand the like.

The driving characteristic acquirer 132 may determine a driving level ofthe occupant on the basis of the driving tendency (for example, a ratioand frequency at which the center of the vehicle M deviates from thecenter of the lane) at the time of traveling on a straight-line orcurved road. For example, the driving characteristic acquirer 132determines that the driving level of the occupant is higher when theratio or frequency at which the center of the vehicle M deviates fromthe center of the lane is lower.

The occupant state acquirer 134 performs an image analysis process foran image captured by the driver monitor camera 70 and detects whether ornot the occupant is in a prescribed state on the basis of an analysisresult. The prescribed state includes, for example, a state in which theoccupant (driver) driving the vehicle M grips the steering wheel, astate in which the occupant monitors the front, and the like. Theprescribed state may include a state in which the occupant is talking toanother fellow passenger, a state in which the occupant is angry, astate in which the occupant is drowsy, and the like. These states can beacquired from the facial movement and expression of the occupant in theimage and the like. The occupant state acquirer 134 may acquire that theoccupant is in a state in which manual driving is possible (or a statein which manual driving is not possible).

The occupant state acquirer 134 may derive a driving concentrationdegree of the occupant (an index value indicating a degree to which theoccupant can concentrate on driving) on the basis of an image analysisresult. For example, the driving concentration degree when the occupantis talking to a fellow passenger is less than that when the occupant isnot talking. When the occupant is yawning and drowsy, the drivingconcentration degree is made lower than the current degree. When thefellow passenger is a child, the occupant state acquirer 134 may reducethe driving concentration degree as compared with when the fellowpassenger is not a child, or the driving concentration degree maydecrease as the traveling time increases. The occupant state acquirer134 may reduce the driving concentration degree when the occupant islooking at the navigation HMI 52 as compared with when the occupant isnot looking at the navigation HMI 52 (when peripheral monitoring is notsufficiently performed). Also, in contrast, the occupant state acquirer134 increases the driving concentration degree, for example, when theoccupant (driver) grips the steering wheel and monitors the front for aprescribed period of time or longer, when there is no fellow passenger,or when the occupant (driver) is not talking to a fellow passenger evenif there is a fellow passenger.

The occupant state acquirer 134 may acquire the presence or absence of afellow passenger, the type of fellow passenger (for example, a child oran elderly person), the number of people, a state, and the like as wellas a state of the occupant (driver) on the basis of the image analysisresult. A part of the above-described information about the driver andthe fellow passenger may be acquired using audio information acquired bythe driver monitor camera 70 as well as images.

The driving controller 140 automatically controls one or both ofsteering or acceleration/deceleration of the vehicle M on the basis of arecognition result of the recognizer 110 and a determination result ofthe determiner 120 to execute driving control. For example, when anoperation for executing at least one of various types of drivingassistance has been received by the HMI 30, the driving controller 140controls the driving operation of the vehicle M by the occupant of thevehicle M or the traveling of the vehicle M on the basis of receivedoperation content.

The driving controller 140 includes, for example, a steering controller142. For example, when the HMI 30 has received an operation indicatingthat the LKAS control is to be executed (an operation by the occupant),the steering controller 142 executes LKAS control for controlling atleast steering of the vehicle M so that the vehicle M travels withoutdeviating from the travel lane. Details of the functions of the steeringcontroller 142 will be described below.

The driving controller 140 may execute control processes of ACC, LCA, anFCW, a CMBS, and the like on the basis of the recognition result of therecognizer 110. For example, the driving controller 140 executes drivingcontrol for causing the vehicle M to follow a preceding vehicle and runwhen an operation of executing ACC control has been received by the HMI30. When an operation of executing LCA control has been received by theHMI 30, the driving controller 140 sets a target position that does notcome into contact with a physical object such as another vehicle in thecenter of a lane that is a lane change destination located in adirection indicated by a direction indicator switch or the like,notifies the occupant of the set target position from the HMI 30, causesthe occupant to execute the steering operation of the vehicle M so thatthe vehicle M is positioned at the target position, or executes steeringcontrol. The driving controller 140 warns the occupant using the HMI 30when there is a possibility of contact with a physical object in an FCWcontrol process and controls emergency stopping of the vehicle M in aCMBS control process.

The HMI controller 150 notifies the occupant of prescribed informationusing the HMI 30 and acquires operation content of the occupant receivedvia the HMI 30. For example, the prescribed information includesinformation related to traveling of the vehicle M such as informationabout the state of the vehicle M and information about driving control.The information about the state of the vehicle M includes, for example,a speed of the vehicle M, an engine speed, a shift position, and thelike. Also, the information about the driving control may include, forexample, information indicating which driving control is being executedor information for asking about whether or not to provide drivingassistance. The prescribed information may include information that isnot related to the traveling control of the vehicle M, such as atelevision program or content (for example, a movie) stored in a storagemedium such as a DVD.

The HMI controller 150 generates an image for allowing the occupant toselect or adjust the steering characteristics to be applied during theexecution of LKAS control, and causes the display device of the HMI 30to display the generated image. The HMI controller 150 may cause animage for suggesting a recommended steering characteristic among aplurality of steering characteristics to be displayed on the basis of aroad situation (for example, a road type, curvature, a distance fromanother vehicle, or a congestion situation) of the travel lane of thevehicle M recognized by the recognizer 110 or the like. The HMIcontroller 150 outputs a selection result and an adjustment result inputby the occupant from the HMI 30 to the acquirer 130 and the steeringcontroller 142 or updates the steering characteristic information 162stored in the storage 160 on the basis of the selection result and theadjustment result.

For example, the HMI controller 150 may generate an image including theabove-described prescribed information and cause the display device 32of the HMI 30 to display the generated image or may generate a soundindicating prescribed information, and cause the generated sound to beoutput from the speaker of the HMI 30.

The travel driving force output device 200 outputs a travel drivingforce (torque) for enabling the vehicle to travel to driving wheels. Forexample, the travel driving force output device 200 includes acombination of an internal combustion engine, an electric motor, atransmission, and the like, and an electronic control unit (ECU) thatcontrols the internal combustion engine, the electric motor, thetransmission, and the like. The ECU controls the above-describedcomponents in accordance with information input from the drivingassistance device 100 or information input from the driving operationelement 80.

For example, the brake device 210 includes a brake caliper, a cylinderconfigured to transfer hydraulic pressure to the brake caliper, anelectric motor configured to generate hydraulic pressure in thecylinder, and a brake ECU. The brake ECU controls the electric motor inaccordance with the information input from the driving assistance device100 or the information input from the driving operation element 80 sothat brake torque according to a braking operation is output to eachwheel. The brake device 210 may include a mechanism configured totransfer the hydraulic pressure generated according to an operation onthe brake pedal included in the driving operation elements 80 to thecylinder via a master cylinder as a backup. The brake device 210 is notlimited to the above-described configuration and may be anelectronically controlled hydraulic brake device configured to controlan actuator in accordance with information input from the drivingassistance device 100 and transfer the hydraulic pressure of the mastercylinder to the cylinder.

For example, the steering device 220 includes a steering ECU and anelectric motor. For example, the electric motor changes a direction ofsteerable wheels by applying a force to a rack and pinion mechanism. Thesteering ECU drives the electric motor in accordance with theinformation input from the driving assistance device 100 or theinformation input from the driving operation element 80 to change thedirection of the steerable wheels. The steering device 220 may apply atorque reaction force so that the steering wheel is rotated in aprescribed direction (or so that the steering wheel is not rotated inthe prescribed direction) for the occupant's driving operation bycontrolling the driving assistance device 100.

[Steering Control]

Next, details of the functions of the steering controller 142 will bedescribed. FIG. 2 is a diagram for describing the function of thesteering controller 142. In the example of FIG. 2 , the vehicle Mtraveling at a velocity VM in a lane L1 in which travel in an X-axisdirection is possible in FIG. 2 is shown. The lane L1 is divided bymarkings LL and RL. In FIG. 2 , a center CL of the lane L1 is shown.

For example, when the execution of the LKAS control has been receivedfrom the occupant, the steering controller 142 controls the steeringdevice 220 and causes the steering wheel ST to output a force (asteering reaction force) opposite to an operation direction of theoccupant in accordance with an amount of deviation (a distance D1) in alateral direction from the center CL (a lane width direction or a Y-axisdirection in FIG. 2 ) so that the center C of the vehicle M passes overthe center CL of the lane L1. In other words, the steering controller142 causes the steering device 220 to output a force for rotating thesteering wheel ST toward the lane center CL from the perspective of thecenter of the vehicle M. Thereby, the vehicle M can be moved to thevicinity of the center CL of the lane L1 and deviation from the lane L1can be suppressed.

Here, the steering controller 142 selects any one of a plurality ofpredetermined steering characteristics in accordance with informationobtained from the occupant of the vehicle M when a reaction force isapplied to the steering wheel ST and controls the steering of thevehicle M so that the vehicle M does not deviate from the travel lane.

FIG. 3 is a diagram for explaining a plurality of steeringcharacteristics according to the embodiment. In the example of FIG. 3 ,the vertical axis represents a reaction force for the steering wheel,and the horizontal axis represents a distance from the center of thetravel lane of the vehicle M. FIG. 4 is a diagram for describing thepositioning of the reaction force in each of the plurality of steeringcharacteristics. In the example of FIG. 4 , the positioning of steeringcharacteristics using a steering freedom degree and an assist feelingdue to the reaction force as index values (biaxial) are shown.

In embodiments, the plurality of steering characteristics included inthe steering characteristic information 162 include at least a firststeering characteristic and a second steering characteristic. The firststeering characteristic is a balanced steering characteristic foremphasizing the balance between the steering freedom degree and theassist feeling as shown in FIG. 4 . The second steering characteristichas a greater assist feeling and a lower operation freedom degree thanthe first steering characteristic. In other words, the second steeringcharacteristic is an aggressive steering characteristic in which thesteering reaction force (the assist amount) is larger than the firststeering characteristic and the steering control is actively performedon the system side. For example, the second steering characteristic hasa larger steering reaction force corresponding to a distance from thecenter of the travel lane of the vehicle M than the first steeringcharacteristic. In relationships between a position of the vehicle M andpositions of the markings on the left and right of the travel lane, asteering reaction force from the marking side to the inside in thesecond steering characteristic may be larger than that in the firststeering characteristic. The type of each steering characteristic is notlimited to the above examples.

The plurality of steering characteristics may include a third steeringcharacteristic that is a characteristic between the first steeringcharacteristic and the second steering characteristic. Thecharacteristic between the first steering characteristic and the secondsteering characteristic is, for example, a characteristic that becomesan intermediate value of the steering characteristics. The steeringreaction force degree of each steering characteristic may beappropriately adjusted by the occupant via the HMI 30. When the steeringcharacteristics are adjusted, it is possible to suppress the reactionforce from becoming excessively small or large by setting an upper limitvalue and a lower limit value for adjustment in advance. The pluralityof steering characteristics may include steering characteristics otherthan the first to third steering characteristics. Hereinafter, thesteering characteristic information 162 will be described as havingfirst to third steering characteristics.

The steering controller 142 may select, for example, a referencesteering characteristic (for example, the first steering characteristic)as an initial setting in advance. The steering controller 142 may selectthe third steering characteristic having an intermediate value betweenthe first and second steering characteristics as an initial setting andthen switch the steering characteristic to either the first steeringcharacteristic or the second steering characteristic in accordance withthe selection of the occupant, the driving characteristic, the occupantstate, or the like.

For example, the steering controller 142 switches the steeringcharacteristic to the steering characteristics selected by the occupantvia the HMI 30 between the first steering characteristic and the secondsteering characteristic described above to control the steering of thevehicle M. The steering controller 142 may perform a switching processat a timing of LKAS control execution indicated through the HMI 30 afterselection or a timing when a prescribed period of time has elapsed fromthe selection without immediately switching the steering characteristicafter the selection by the occupant. Thereby, the reaction forceswitching can be suppressed in the middle of steering control and saferdriving assistance can be performed.

The steering controller 142 may control the steering of the vehicle M byselecting any one of the plurality of steering characteristics on thebasis of the driving characteristics of the occupant acquired by thedriving characteristic acquirer 132. For example, the steeringcontroller 142 controls the steering of the vehicle by selecting thesecond steering characteristic when the driving characteristic is acharacteristic of the vehicle being likely to deviate from the travellane and selecting the first steering characteristic as the referencecharacteristic otherwise. Characteristics of the vehicle being likely todeviate from the travel lane include, for example, drivingcharacteristics of a tendency for a ratio and frequency at which thevehicle M deviates from the center of the lane to be high (above athreshold value), a tendency for the timing of the steering operation tobe later (or earlier) than the reference range, and a tendency for thesteering amount to be smaller (or larger) than the reference amount.

For example, the steering controller 142 controls the steering of thevehicle M by selecting the second steering characteristic when thedriving characteristic indicating that the driving level of the occupantis low (the level is less than the threshold value) is obtained andcontrols the steering of the vehicle M by selecting the first steeringcharacteristic when the driving characteristic indicating that thedriving level of the occupant is high (the level is greater than orequal to the threshold value) is obtained.

The steering controller 142 may select any one of the plurality ofsteering characteristics on the basis of the occupant state acquired bythe occupant state acquirer 134 to control the steering of the vehicleM. For example, the steering controller 142 controls the steering of thevehicle by selecting a second steering characteristic when the occupantstate is a state in which the vehicle is likely to deviate from thetravel lane and selecting the first steering characteristic as thereference characteristic otherwise. States in which the vehicle islikely to deviate from the travel lane include, for example, a state inwhich the occupant is talking to another fellow passenger, a state inwhich the occupant is angry, a state in which the occupant is drowsy,and the like. States in which the vehicle is likely to deviate from thetravel lane may include a case where a fellow passenger is a child.

For example, on the basis of the occupant state, the steering controller142 controls the steering of the vehicle M by selecting the secondsteering characteristic when the driving concentration degree of theoccupant is low (when the driving concentration degree is less than thethreshold value) and controls the steering of the vehicle M by selectingthe first steering characteristic when the driving concentration degreeof the occupant is high (when the driving concentration degree isgreater than or equal to the threshold value).

As described above, it is possible to perform reaction force controlsuitable for a preference or state of the occupant by selecting asteering characteristic in accordance with the information obtained fromthe occupant of the vehicle M (for example, a steering characteristicselection result of the occupant, the driving characteristic of theoccupant, and the occupant state) to control the steering of the vehicleM. Therefore, appropriate lane keeping assistance for the occupant canbe provided.

The steering controller 142 may select any one of a plurality ofsteering characteristics on the basis of a road situation around thevehicle M recognized by the recognizer 110 in addition to informationobtained from the occupant of the vehicle M. For example, the steeringcontroller 142 may select the second steering characteristic when thetravel lane of the vehicle M is an expressway and select the firststeering characteristic because there is a distance from a lane end whenthe width of the travel lane is wide (when the width of the travel laneis greater than or equal to a prescribed width). For example, thesteering controller 142 selects the first steering characteristic whenthe curvature of the travel lane is less than a threshold value andselects the second steering characteristic when the curvature of thetravel lane is greater than or equal to the threshold value.Furthermore, the steering controller 142 may select the steeringcharacteristic in accordance with a velocity of the vehicle M or thelike.

The steering controller 142 controls the steering of the vehicle M byswitching the steering characteristic to the steering characteristicselected in accordance with a higher-priority condition on the basis ofpriorities for various types of conditions for selecting theabove-described steering characteristics (a condition for informationobtained from the occupant of the vehicle M and information of a roadsituation). The priority may be preset or set to be variable by theoccupant. By adjusting the priority according to the occupant's setting,it is possible to select the steering characteristic corresponding tothe occupant's preference.

The above-described selection of steering characteristics by thesteering controller 142 may be performed during execution as well asbefore or after execution of LKAS control. Therefore, if the steeringcharacteristic is switched between the first steering characteristic andthe second steering characteristic during manual driving by theoccupant, because the reaction force control will be different frombefore, discomfort is likely to be given to the occupant. Therefore,when the steering characteristic is switched between the first steeringcharacteristic and the second steering characteristic, the steeringcontroller 142 may temporarily perform steering control of the vehicle Mbased on the third steering characteristic. The term “temporarily” mayindicate, for example, a prescribed period of time after a process ofdeciding to switch the steering characteristic to one or the other ofthe first and second steering characteristics, a period until thevehicle M travels a prescribed distance after the steeringcharacteristic is switched to the third steering characteristic, or aperiod until a prescribed period of time of steering control based onthe third steering characteristic has elapsed. The steering controller142 may cause the HMI 30 to output which steering characteristic iscurrently being used for steering control by the HMI controller 150.Thereby, it is possible to suppress discomfort from being given to theoccupant when the steering characteristic is switched and execute moreappropriate lane keeping control.

The steering controller 142 may select any steering characteristic amongthe plurality of steering characteristics and control the steering ofthe vehicle M in accordance with a change in the state of the drivingcontrol in operation. For example, the steering controller 142 executessteering control based on the first steering characteristic for aprescribed period of time immediately before or immediately after ALCcontrol in a state in which the steering of the vehicle M is beingcontrolled in the second steering characteristic during the execution ofthe LKAS control and when the ALC control is started according to anoperation by the occupant on the direction indicator switch.

[Processing Flow]

FIG. 5 is a flowchart showing an example of a flow of a process executedby the driving assistance device 100 of the embodiment. In the processof FIG. 5 , the lane keeping control process among various types ofprocesses executed by the driving assistance device 100 will be mainlydescribed. The process of FIG. 5 may be performed repeatedly atprescribed cycles or at prescribed timings. At the beginning of theprocess of FIG. 5 , it is assumed that an instruction for causing thevehicle M to execute LKAS control is received by the occupant via theHMI 30.

In the example of FIG. 5 , the recognizer 110 recognizes a surroundingsituation of the vehicle M (step S100). Subsequently, the drivingcontroller 140 executes LKAS control on the basis of a recognitionresult of the recognizer 110 (step S102). During the execution of LKAScontrol, the processing of step S101 is executed all the time.

Subsequently, the determiner 120 determines whether or not the vehicle Mtends to deviate from the travel lane on the basis of a position (adeviation amount) of the vehicle M with respect to the center of thetravel lane (step S104). When it is determined that the vehicle M tendsto deviate from the travel lane, the steering controller 142 selects anysteering characteristic in accordance with information obtained from theoccupant of the vehicle M among the plurality of steeringcharacteristics (step S106) and executes reaction force control on thebasis of the selected steering characteristic (step S108). After the endof step S108 or when it is determined that the vehicle does not tend todeviate from the travel lane in the processing of step S104, the drivingcontroller 140 determines whether or not to end LKAS control (stepS110). For example, the driving controller 140 determines to end theLKAS control when an ALC control execution instruction or an LKAScontrol end instruction, an operation of a direction indicator switch,or the like has been received from the HMI 30 and determines not to endthe LKAS control otherwise. When it is determined not to end that LKAScontrol, the process returns to step S104. When it is determined to endthe LKAS control, the process of the present flowchart ends.

According to the embodiment described above, the driving assistancedevice 100 includes the recognizer 110 configured to recognize asurrounding situation of the vehicle M; and the steering controller 142configured to control at least steering of the vehicle so that thevehicle M travels in a travel lane when the vehicle M tends to deviatefrom the travel lane on the basis of a recognition result of therecognizer 110, wherein the steering controller 142 controls thesteering of the vehicle M by selecting any steering characteristic inaccordance with information obtained from an occupant of the vehicle Mamong a plurality of steering characteristics, thereby providingappropriate lane keeping assistance for an occupant. Also, it ispossible to contribute to the development of a sustainabletransportation system.

Specifically, according to the embodiment, it is possible to furtherimprove the marketability of the LKAS system by switching the steeringcharacteristic between a weak steering reaction force characteristic (afirst steering characteristic) in which the steering is entrusted to theoccupant to some extent and a strong steering reaction forcecharacteristic (a second steering characteristic) for controllingsteering on the system side during execution of the LKAS control on thebasis of instruction content and driving characteristics of theoccupant, a state of the occupant, and the like and by providing anoptimal steering reaction force during the LKAS control according to theoccupant's preference.

The embodiment described above can be represented as follows.

A vehicle assistance device including:

-   -   a storage medium storing computer-readable instructions; and    -   a processor connected to the storage medium,    -   the processor executing the computer-readable to:    -   recognize a surrounding situation of a vehicle;    -   control at least steering of the vehicle so that the vehicle        travels in a travel lane when the vehicle tends to deviate from        the travel lane on the basis of a recognition result; and    -   control the steering of the vehicle by selecting any steering        characteristic in accordance with information obtained from an        occupant of the vehicle among a plurality of steering        characteristics.

While preferred embodiments of the invention have been described andillustrated above, it should be understood that these are exemplary ofthe invention and are not to be considered as limiting. Additions,omissions, substitutions, and other modifications can be made withoutdeparting from the spirit or scope of the present invention.Accordingly, the invention is not to be considered as being limited bythe foregoing description, and is only limited by the scope of theappended claims.

What is claimed is:
 1. A driving assistance device comprising: arecognizer configured to recognize a surrounding situation of a vehicle;and a steering controller configured to control at least steering of thevehicle so that the vehicle travels in a travel lane when the vehicletends to deviate from the travel lane on the basis of a recognitionresult of the recognizer, wherein the steering controller controls thesteering of the vehicle by selecting any steering characteristic inaccordance with information obtained from an occupant of the vehicleamong a plurality of steering characteristics.
 2. The driving assistancedevice according to claim 1, wherein the plurality of steeringcharacteristics include at least a first steering characteristic and asecond steering characteristic having a larger steering reaction forceof the vehicle than the first steering characteristic.
 3. The drivingassistance device according to claim 2, wherein the second steeringcharacteristic has a larger steering reaction force corresponding to adistance from a center of the travel lane of the vehicle than the firststeering characteristic.
 4. The driving assistance device according toclaim 1, wherein the steering controller controls the steering of thevehicle by switching the steering characteristic to the steeringcharacteristic selected by the occupant among the plurality of steeringcharacteristics.
 5. The driving assistance device according to claim 1,further comprising a driving characteristic acquirer configured toacquire driving characteristics of the vehicle from the occupant of thevehicle, wherein the steering controller controls the steering of thevehicle by selecting any steering characteristic among the plurality ofsteering characteristics on the basis of the driving characteristicsacquired by the driving characteristic acquirer.
 6. The drivingassistance device according to claim 2, wherein the plurality ofsteering characteristics further include a third steering characteristicfor controlling the steering of the vehicle according to a steeringreaction force between the steering reaction force based on the firststeering characteristic and the steering reaction force based on thesecond steering characteristics, and wherein the steering controllertemporarily controls the steering of the vehicle according to the thirdsteering characteristic when the steering characteristic is switchedbetween the first steering characteristic and the second steeringcharacteristic.
 7. The driving assistance device according to claim 1,further comprising an occupant state acquirer configured to acquire astate of the occupant of the vehicle, wherein the steering controllercontrols the steering of the vehicle by selecting any steeringcharacteristic among the plurality of steering characteristics on thebasis of the occupant state acquired by the occupant state acquirer. 8.The driving assistance device according to claim 1, wherein the steeringcontroller controls the steering of the vehicle by selecting anysteering characteristic among the plurality of steering characteristicson the basis of information obtained from the occupant of the vehicleand a road situation near the vehicle recognized by the recognizer.
 9. Adriving assistance method comprising: recognizing, by a computer, asurrounding situation of a vehicle; controlling, by the computer, atleast steering of the vehicle so that the vehicle travels in a travellane when the vehicle tends to deviate from the travel lane on the basisof a recognition result; and controlling, by the computer, the steeringof the vehicle by selecting any steering characteristic in accordancewith information obtained from an occupant of the vehicle among aplurality of steering characteristics.
 10. A computer-readablenon-transitory storage medium storing a program for causing a computerto: recognize a surrounding situation of a vehicle; control at leaststeering of the vehicle so that the vehicle travels in a travel lanewhen the vehicle tends to deviate from the travel lane on the basis of arecognition result; and control the steering of the vehicle by selectingany steering characteristic in accordance with information obtained froman occupant of the vehicle among a plurality of steeringcharacteristics.